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ABSTBACT 

The position that the National Assessment of 
Educational progress (NAEP) provides a data base that is compatible 
with various definitions of literacy and standards of competencies 
and can make a contribution to cur understanding of the complex and 
controversial problem of literacy is supported in this paper. The 
paper notes that since the naeP has surveyed a national sample of 
four age groups over a period of years since 1971, it is well suited 
for looking at a broad sample and providing longitudinal information. 
Other advantages of the naSP for studying literacy that are cited are 
its comprehensiveness across many areas of study and functional 
skills* its breadth vithin single content areas, its covering of 
affective and background variables as well as achievement, and its 
ability to respond to new emphases, such as the proposed 1979*80 
reading and literature assessment that has new exercises emphasizing 
comprehension of various kinds of texts. The use of the naep data 
base for the assessment of functional literacy conducted by KAEP in 
197*1 and 1975 for the National Eight to Read program is discussed as 
an example of the use of the naep data base* (HKB) 
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Introduction 

Any effort directed toward extending our understanding of literacy 
in the United States must accept as a given that no single study* how- 
ever ambitious or broad, is going to provide more thsn a fa* of the many 
answers we need. The problems of educational literacy are complex and 
far-reaching. Up to this point, the literature directed toward this sub- 
ject has been diverse and often contradictory. There are dozens of major 
and countless minor definitions of literacy. Moreover, over the last 
decade we have see the definitions shift and change. In addition to the 
conceptual controversy related to literacy, there are equally numerous 
methodological issues concerning the ways literacy is measured and the 
types of standards which are applied. Indeed, suspicions are raised when 
14% of professional and managerial people (who by societal standards would 
normally be considered "successful") have failed to pass a literacy test 
(Fisher, 1978). It is not the purpose of this paper to review all of the 
conceptual and methodological issues related to literacy. These have been 
thoroughly discussed by several authors (see Nafziger, Thompson* Hiscox 1 
Owen, 1975; Kirsch & Guthrie, 1978; Fisher, 1978). Instead, we acknowledge 
the fact that literacy is a con^lex and controversial problem and will show 
how the National Assessment of Educational Progress (NAEP) can make a modest 
contribution to extending our understanding of this phenomenon. 

Characteristics of the National Assessment 

The National Assessment* in its goals and design, is well suited for 
looking at broad, multifaceted educational and social issues, including 
literacy. The primary purpose of the National Assessment is the 
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"assessment of the performance of children and young adults in the basic 
skills of reading, mathematics, and communication" {Public Law 95-561, 
1978). As a part of this mandate, NAEP shall "report periodically on 
changes in knowledge and skills of such students over a period of time 
[and] conduct special assessments of other educational areas » as the need 
for additional national information arises" {Public Law 95-561, 1978). 
Current specifications for the continuation of the project delineate 
NAEP's role: 

NAEP should not be conceived of as a national test. It is 
rather, an assessment designed to improve the practice of educa- 
tion in state and local educational agencies , with this In mind, 
applicants must avoid designing and implementing the project as 
though it were to lead to the development of federal tests, cur- 
ricula, or standards {National Institute of Education, 1979, p. 12). 

In its implementation, the National Assessment takes the fom of a 
national sample survey of four age groups: 9-year-olds, 13-year-olds, 17- 
year-olds, and a young adult group which includes 26 to 35-year-olds. The 
last two age groups are probsbly the natural focus for the stuflly of lit- 
eracy; although data on earlier age groups might shed some light on symp- 
toms and future trends. Ten educational content areas have been surveyed;^ 
most have been surveyed at two or even three points in time. In addition* 
special probe areas such as basic life skills, health, consumerism and 
energy have been assessed. Each year one or more content areas are sur- 
veyed. Approximately twelve booklets of assessment items {each booklet 
comprising approximately 35 minutes of testing) are administered to each 
age group. An individual takes only or.f set of itents; a separate national 
sample of approximately 2,500 individuals is used for each booklet. One 

Hhe ten content areas include art, career and occupational develop- 
ment, citizenship, literature, mathematics, music, reading, science, social 
studies, and writing. ^' 



of the unique characteristics of NAEP is its multi-stage, 5^ 'ied, prob 
ability sampling pi^ocedure which gives accurate estimates c mal an^^ 

group performance at one or more points in time. 

Because the National Assessment focuses on performanca of it- 
students and young adults (rather than individuals) withou : ref :o 
specific standards, it could be argued NAEP is of limited useful: or 
a study of literacy, However, within this framework, we f that tAEP 
offers some unique capabilities for exploring the problems of literacy. 
These capabilities reside in the breadth of coverage and longitudinal 
aspects of the data base. This paper will explain the advantages of the 
National Assessment data base and its past and potential contributions to 
an understanding of the problems of literacy. 

Advantages of the National Assessment for Studying Llterac_y 

One of the unique aspects of the National Assessment survey is its 
comprehensiveness. This breadth is apparent in the number of areas it 
addresses* the variety of assessment tasks within an area* and the inclu- 
sion of affective and background information. The National Assessment 
covers the full range of contents which are the focus of our nation's 
schools, as well as areas of emerging importance; for example, energy and 
conr;umerism* Within this scope, MAEP can accomnodate a large range of 
definitions of literacy. The MAEP data base is applicable to approaches 
to literacy which focus on the most basic aspects of the production and use 
of written material. It can encompass basic computational skills as well. 
The survey includes a variety of basic life skills such as reading traffic 
signs or filling out a job application fonu* Finally, for those who take 
a broad view of literacy, the assessment offers an opportunity to consider 



concepts of literacy in areas such as the fine arts* science or citizen- 
ship. Thus* the National Assessment is not limited to a single approach 
to literacy. It provides opportunities for exploring many different 
alternatives and the chance to compare and contrast several approaches to 
literacy. 

Another important aspect of the coverage offered by the National 
Assessment is the breadth represented within a single content area. For 
example* the area of writing includes a variety of writing tasks—descrip- 
t1ve> persuasive and creative. It assesses the ability of students to 
achieve their writing purpose as well as their skills in spelling and 
mechanics. Thus> a full range of skills within an area may be considered 
in an assessment of literacy. This seems particularly important for 
those who take a situational approach to literacy by defining literacy for 
a factory worker* for example* in a different way than for a high school 
teacher. It also allows us to analyze various pieces of the problem. For 
example* we can find out that U-year-olds can communicate a simple 
explanatory message but sometimes use awkward sentences. 

The National Assessment is not just an achievement survey. It also 
assesses affective and background variables. For example* the Mathematics 
Assessment includes valuable information about students' attitudes toward 
mathematics* previous instruction* and experience with Culculators^-all 
perhaps equally as important as achievement measures with respect to an 
individual's ultimate ability to cope with basic computational tasks. 
Across the content areas* basic socioeconomic and demographic information 
is collected from all students and schools in the sample. In exploring a 
phenomenon as complex as literacy* no matter how defined* it seems advan- 
tageous to include other factors besides achievement. 
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In addition to the content breadth of the National Assessment, a 
second unique advantage of the NAEP data base is its longitudinal nature. 
As indicated earlier, the purpose of NAEP is not to compare individuals 
against standards of performance. The only benchmarks related to literacy 
which NAEP provides are national and group average perfonnance on ite 
and groups of items. This does not preclude the possibility of other 
groups or individuals selecting items which fit their definition of lit- 
eracy and setting standards which reflect their concept of competence. 

There is, however, currently much controversy among measurement 
experts about the advisability and feasibility of setting competency stan- 
dards, especially so for "life skill" or "survival" competencies where it 
is necessary to consider perfonnance on criterion measures of life success 
(see Bunda & Sanders, 1979; Hambleton, Powell & Eignor, 1979; also the 
entire issue of the Journal of Educational Measurement— Vol . 15, No, 2, 
1978), In light of this controversy, the ability of NAEP to measure trends 
over time provides an important methodological alternative in the study of 
literacy. By reporting whether perfonnance goes up or down over time, NAEP 
can avoid the standard-setting dilemna, while still being able to chart the 
progress of literacy attainment in the nation- 

Thus, National Assessment provides several unique capabilities for 
exploring the problems of literacy in the United States, It provides a 
data base which is compatible with a variety of views of literacy. It does 
not provide standards; however, standards may be applied from outside 
sources* Furthermore, it indicates changes in performance over time* To 
examine these capabilities further, we will provide ar description of how 
the National Assessment has been used to furnish the Right to Read Program 
with national trend data on functional literacy. We will then suggest some 



of the potential value of NAEP data for future literacy studies, especially 
for those studies which might be based on the 1979-80 Reading/Literature 
Assessment. 

The Mini-Assessment of Functional Literacy 

In order to obtain more concrete evidence on the status and changes 
in functional literacy, the National Right to Read Program awarded a grant 
to the Education Coninission of the States for NAEP to conduct in 1974 and 
1975 a Mini-Assessment of Functional Literacy (MAFL). The major goal of 
MAFL was to detennine the extent of functional literacy among all 17-year- 
old students in the United States and among various subgroups of 17-year- 
old students. 

A panel of reading specialists appointed by Right to Read selected 
86 exercises representing the types of reading materials encountered In 
everyday life; for example, signs, maps, graphs, fomrs, news articles, 
dictionaries and telephone directories. (For a more complete description 
of the development and results of MAFL, see Gadway & Wilson, 1976.) Of 
the 86 exercises, 64 had been used in National Assessment's 1971 regular 
reading assessment of 17-year-old students. This subset of exercises, 
called the "truncated" MAFL, provided data for three time pointS"1971, 
1974 and 1975. Each MAFL exercise was classified into one of the five 
following reading skills: (1) understanding word meanings, (2) gleaning 
significant facts, (3) comprehending main ideas and organization, (4) draw- 
ing inferences, and (5) reading critically. 

The results of MAFL were presented in relation to several different 
standards including: 

.6.8 
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1, The highest expected level of performance (HELP)"This level was 
defined as the best performance one can reasonably expect based on the 
performance of a group of known superior readers who have achieved above 
the ninetieth percentile on a standardized reading test. Here, the per- 
centage of students responding correctly to an exercise was adjusted to a 
percentage of the superior readers' performance on that exercise* For 
example, If 70% of 17-year-old students answered a given exercise correctly 
and 90% of the superior readers answered the same exercise correctly, the 
percentage of success Is adjusted to 77,81% (I.e. > 70% divided by 90%, and 
multiplied by one hundred). The HELP standard was chosen as 100% and If 
the adjusted percentage of success on an exercise was below this standard. 
It represented a shortfall from this criterion. 

2. The minimally adequate perfonnance standard (MAP)--This was the 
lowest level above which 17-year-old students were considered functionally 
literate. Right to Read determined that a 17-year-old student must be 
able to answer at least 75% of the exercises to be considered functionally 
literate. 

Average performance on the truncated MAFL set of 64 exercises is 
depictfid for 1971, 1974 and 1975 in Exhibit 1* Performance increased 
nationally from 83*7% to 85*9% between 1971 and 1975. All twenty popula- 
tion groups for which NAEP normally reports results (nationally and by 
geographic region, sex, race, parental education, and size and type of com- 
munity) showed some Improvement between 1971 and 1975* For fourteen of 
these twenty groups, this improvement in average perfonnance was statisti- 
cally significant at the 0,05 probability level. 

On the complete MAFL set of 86 exercises administered in 1974 and 
again in 1975, 17-year-old students did best on reading tasks involving 
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understanding word meanings In context, and they did least well on drawing 
inferences and reading critically. On the complete MAFL In 1974, the mean 
percentage correct was 87.5%, and this percentage remained virtually 
unchanged at 87.6% in 1975. When these two percentages were adjusted by 
the HELP procedures, the percentages Increased to 91.2% In 1974 and to 
91.4% in 1975. These percentages fell somewhat short of the HELP crite- 
rion of 100%. At the national level on the 1975 complete MAFL, 87.4% of 
17-year-old students attained the MAP standard of at least 75% correct. 
Certainly a stu<(y such as MAFL Is subject to criticism. {Critical 
reviews of MAFL may be found In Harnischfeger & Wiley, 1976i Fisher, 1978; 
Kirsch & Guthrie, 1978.) However, wha-t seems Important is the fact that 
the National Assessment, which Is broad In purpose and scope, was able to 
respond to some very specific concerns related to functional literacy. As 
a result of this effort, we were able to provide some unique trend Infor- 
mation about a carefully sampTed population of 17-year-old students. It is 
this type of activity which we would like to encourage further and the 
following section will suggest some potential contributions of NAEP for the 
future. 

The 1979-80 ReadlnflZLIterature Assessment 

The 1979-80 Reading/Literature Assessment will provide additional 

Information which could be used for exploring the problems of literacy. 

This survey has two distinct aspects: (1) the reassessment of reading and 

literature exercises from previous assessments for determining change, and 

(2) the administration of newly-developed exercises for establishing a new 

baseline for future Change analysis. The old reading exercises will show 

change across three time po1ntS"1971, 1975 and 1980, The new exercises 

* 
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win indicate performance at one point In time on a range of activities. 
The new development reflects an Integrated approach to reading and litera- 
ture. It focuses about eqiially on expository and literary materials. The 
passages are all drawn from actual published material and represent what 
students typically read In school and out of school. They range in diffi- 
culty and length from a three-sentence news feature to a complete short 
story of almost 2.000 wordl. 

A cOTiplete list of categories of reading materials is provided In 
Exhibit 2. Included among the list is a category which Is specifically 
identified as "functional." These passages represent reading matter which 
students are faced with In many everyday activities. Examples from this 
category Include a want ad* a recipe* various kinds of labels, application 
forms and articles with various types of utilitarian Information. In addi- 
tion, several passages provide exairqiles of commonly used persuasive ar^d 
critical language. These include several advertisements > reviews and an 
editorial. Finally, several of the study skills exercises utilize func- 
tional materials. Including graphs on energy use and consumer spending, 
train and bus schedules, a weather map, and a yellow page from the tele- 
phone book. 

The largest portion of the new development focuses on comprehension 
tasks and these too reflect diversity and a range of difficulty. They In- 
clude: (1) comprehending words In context and lexical relationships, such 
as actor, action and object* (Z) comprehending proposltlonal relationships* 
such as cause, time and Instrument* and (3) comprehending textual rela- 
tionships, such as main Idea* purpose* theme* character* mood and tone. An 
outline of the objectives is provided In Exhibit 3. 
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Exhibit 2. Categories of 1979-80 Reading/Literature passages 



1.0 Expository Materials 

1.1 Science 

1.2 Social studies 

1.3 People 

1 A Leisure 

1.5 Functional 

1.6 Persuasive and critical 

2.0 Literary Materials 

2.1 Stories 

2.2 Other literary prose 

2.3 Po^s 
2A Plays 



Exhibit 3. Outline of the 1979-80 Reading/Literature Objectives 

1.0 Values Reading and Literature 

1-1 Values reading and literature as a source of enjoyment 

1.2 Values reading and literature as a means for gaining In- 
formation and solving problems 

1.3 Values reading and literature as a means for increasing 
self-understanding and understanding of other people 

1.4 Appreciates the role of reading and literature in sustain 
ing and changing culture 

2.0 Comprehends Written Works 

2.1 Comprehends words in context and lexical relationships 

2.2 Comprehends propositions and proposltlonal relationships 

2.3 Comprehends textual relationships 

3.0 Responds to Written Works 

3.1 Responds to written works emotionally and personally 
'3.2 Responds to written works Interpretatlvely 
3.3 Responds to written works analytically 
3 A Responds to written works evaluatively ■ 

4.0 Applies Study Skills 

4.1 Can use«fliifarts» graphs and maps 

4.2 Can use parts of a book 

4.3 Can use library and reference materials 

4.4 Can use study techniques 
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In addition to comprehension Items, part of the assessment focuses 
on students' values regarding reading and literature* students' responses 
to written workSi and students' abilities to use various study skills. 
Finally, background Information Is collected about students' reading ex- 
perience and other relevant factors* such as television viewing, time 
spent on homework and bilingual experience. 

The variety In stimulus materials* comprehension tasks and other In- 
formation on student performance, attitudes and background In the 1979-80 
Reading/Literature Assessment should provide a fruitful data base for ex- 
ploring the problems of literacy. Of special relevance would be an In- 
depth study of the NAEP change data, perhaps focusing on trends In func- 
tional literacy achievement spanning the 1971-1980 time period. Tor 
Instance, there was a national decline between 1971 and 1975 In "higher 
level" reading skills of 13- and 17-year-olds, but during that same time 
period, scores did not decline on the more "basic" reading skills required 
to function in society. Will the 1979-80 assessment show continuations 
of those trends? Will 9-year-olds continue to show improvement in all 
areas? Will the Improvement of reading skills of 9-year-old blacks In 197S 
turn up again In their 13-year-old counterparts In 1979? Will a more 
detailed analysis of the performance data and student background Information 
provide better explanations for the observed trends? 

The newly-developed materials will likewise provide a rich data base 
for studying literacy. For those who are Interested In looking at a con- 
tinuum of text difficulty* there would be an opportunity to explore perfor- 
mance across a range of materials. How well can students comprehend a 
variety of functional materials from simple instructions to complex poli- 
tical rhetoric? It would be possible to focus on particular types of 
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comprehension tasks. How well do students comprehend simple, explicit 
messages? How well do they comprehend Implied messages? Various atti- 
tudinal and background data could be used In conjunction with performance 
data to explore contributory or related factors. How do students' values, 
self-concept as a reader and family background affect comprehension? 

We see from our past experience with MAFL and anticipate with the 
addition of the 1979-80 Reading/Literature Assessment that the National 
Assessment can make an important contribution to understanding the problems 
of literacy, NAEP provides a data base which Is compatible with various 
definitions of literacy and standards for competence, including a measure 
of trends. Certainly NAEP cannot provide us with all the data we need 
about this critical educational and social Issue. However, an exploration 
of various sources of data, including NAEP, is likely to push us toward a 
better understanding of this complex problem. The National Assessment is 
interested in cooperating with individuals and organizations in this impor- 
tant effort. 
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